Controlling Robotic Motion of Camera

ABSTRACT

Among other disclosed subject matter, a system includes a first camera generating a live image of a scene, the first camera configured for being placed in a plurality of locations by robotic motion. The system includes a handheld device that includes a display device for continuously presenting the live image, wherein movement of the handheld device causes the handheld device to generate an output that controls the robotic motion.

TECHNICAL FIELD

This document relates to controlling robotic motion of a camera.

BACKGROUND

Digital or film cameras can be used for the production of motionpictures. Sometimes, these cameras or support systems for the camerasare robotically operated, generally by a technician under the directionof a director of photography. Computer-based systems employing graphicsengines can be used for film or video production. Virtual settings orcharacters are sometimes created by designers and animators forinclusion in film or video with real-world settings or characters.

SUMMARY

The invention relates to controlling robotic motion of a camera.

In a first aspect, a system includes a first camera generating a liveimage of a scene, the first camera configured for being placed in aplurality of locations by robotic motion. The system includes a handhelddevice that includes a display device for continuously presenting thelive image, wherein movement of the handheld device causes the handhelddevice to generate an output that controls the robotic motion.

Implementations can include any or all of the following features. Thehandheld device can include a tablet device having a generally flatconfiguration with the display device located on a main face of thetablet device. The handheld device can include an inertia sensor and theoutput controlling the robotic motion of the first camera can begenerated using the inertia sensor. The handheld device can include anoptical sensor and the output controlling the robotic motion of thefirst camera can be generated using the optical sensor. The handhelddevice can include a magnetic sensor and the output controlling therobotic motion of the first camera can be generated using the magneticsensor. The system can further include a plurality of second camerasmounted in a fixed arrangement around an area where the handheld deviceis to be used, the second cameras registering the movement of thehandheld device, wherein the output controlling the robotic motion ofthe first camera is generated using the second cameras. The system canfurther include an automated crane on which the first camera is mounted,the automated crane configured for receiving the output and placing thefirst camera in any of the plurality of locations by the robotic motion.The first camera can be associated with a first set where the scenetakes place, the first camera can generate the live image from the firstset, and the system can further include a second camera associated witha second set, the second camera configured to generate another liveimage from the second set; wherein the output that controls the roboticmotion of the first camera can also control the second camera. Thesecond set can be a virtual set generated by a computer and the secondcamera can be a virtual camera generated by the computer.

In a second aspect, a computer-implemented method for controlling acamera includes receiving location information regarding a handhelddevice carried by an operator, the handheld device including a displaydevice for continuously presenting a live image from a first cameraconfigured for being placed in a plurality of locations by roboticmotion. The method includes calculating a new location for the firstcamera using the location information. The method includes placing thefirst camera in the new location by the robotic motion while presentingthe live image on the display device.

Implementations can include any or all of the following features. Themethod can further include identifying an initial location of the firstcamera; and calculating a path for the first camera from the initiallocation to the new location, the first camera to traverse the path bythe robotic motion. The method can further include identifying at leastone obstruction between the initial location and the new location; andtaking the obstruction into account in calculating the path so that theobstruction does not interfere with the first camera. The first cameracan be associated with a first set and the obstruction can be identifiedusing a model of the first set. Calculating the path can includedetermining a movement of the handheld device based on the locationinformation; scaling the movement into a scaled movement according to atleast one predefined parameter; and determining the path so that thefirst camera undergoes the scaled movement. The method can furtherinclude causing the first camera to undergo a predefined movementspecified based on path information; identifying an aspect of thepredefined movement to be modified using the handheld device; modifyinga parameter using the location information, the parameter correspondingto the aspect of the predefined movement; and recording the modifiedparameter in association with the path information such that themodified parameter is taken into account in a subsequent use of the pathinformation. The method can further include registering an output fromthe handheld device generated by the operator moving the handheld devicebefore the first camera undergoes the predefined movement, the operatormoving the handheld device to specify the predefined movement; andrecording the path information based on the registered output before thefirst camera undergoes the predefined movement. The method can furtherinclude receiving the path information before the first camera undergoesthe predefined movement, the path information received from anapplication program configured for specifying robotic camera control.The first camera can be associated with a first set and generate thelive image from the first set, and the method can further includeassociating a second camera with a second set, the second cameraconfigured to generate another live image from the second set; placingthe second camera in another location based on the location information,while presenting at least portions of the live image and the other liveimage on the display device. The second set can be a virtual setgenerated by a computer and the second camera is a virtual cameragenerated by the computer.

In a third aspect, a system includes a camera generating a live image ofa scene. The system includes an automated crane on which the camera ismounted, the camera configured for being placed in a plurality oflocations by robotic motion using the automated crane. The systemincludes a handheld device that includes a display device forcontinuously presenting the live image. The system includes a positiondetector generating an output that controls the robotic motion based onmovement of the handheld device, wherein the robotic motion causes thecamera to emulate the movement of the handheld device while the liveimage is presented on the display device.

Implementations can provide any, all or none of the followingadvantages. Improved control of camera robotic motion can be provided.An improved user interface to a camera motion control system can beprovided. A handheld device that affects camera location or motion whileallowing an operator to see the resulting camera output can be provided.

The details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features and advantages willbe apparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 shows an example system that can provide remote control to arobotic camera.

FIG. 2 shows examples of device movement that can be used to controlrobotic camera movement.

FIG. 3 shows an example system that can provide remote control torobotic and/or virtual cameras.

FIG. 4 shows an example of a process for placing a camera in a location.

FIG. 5 is a block diagram of a computing system that can be used inconnection with computer-implemented methods described in this document.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG. 1 is a diagram of an example system 100 for providing remotecontrol to a robotic camera. In some implementations, the system 100 canbe used for production of image contents such as video or motionpictures. Robotic safeguards can be put into place when implementing thesystem 100, such as mechanical guards, presence sensing devices, tripdevices, emergency stop switches, work envelope limit stops, and otherrobotic safeguards, but these are not explicitly shown in the figure forclarity. In examples below, it will be described that an operator cancontrol a robotically maneuvered camera while viewing essentially realtime feedback of an image captured by the camera.

The system 100 in this example includes a camera 102 that can be mountedon an automated robotic crane 104. Any kind of camera can be used, suchas a digital camera or a camera recording on light-sensitivephotographic film, to name a few examples. Any kind of robotic crane canbe used, such as a mechanized arm with one or more articulated jointsand that can be controlled by electronic input, such as from a computer.In some implementations, the camera 102 and the robotic crane 104 can bepositioned on a dolly 106 that can be caused to move along a dolly track108. In some implementations, the dolly 106 can move without track, forexample when surface conditions are favorable or when the dolly canotherwise be effectively controlled. The camera 102, the robotic crane104, and the dolly 106 can be placed in any of a variety of positionsrelative to a scene 110 to achieve a variety of camera movements andangles, for example. In some implementations, the camera 102 can bepositioned to generate a live image of the scene 110, the camera 102configured for being placed in a plurality of locations by roboticmotion.

The system 100 in this example also includes a handheld device 112 thatincludes a display device 114. In some implementations, the handhelddevice 112 can continually present a live image generated by the camera102, and movement of the handheld device 112 can control the roboticmotion of the camera 102, and/or the crane 104, and/or the dolly 106. Insome implementations, the handheld device 112 can be or include a tabletdevice having a generally flat configuration with the display device 114located on a main face thereof. The tablet device can include suitablecomponents for performing its functions, such as a memory and/or harddrive for storing information such as software; a processor forexecuting instructions regarding the operation of the device; a driveror other component interfacing with the display device 114; and a bus orother connector by which two or more of the components can interact witheach other.

The handheld device 112 can be controlled by an operator 116, such as acinematographer, for example. In some implementations, the operator 116can move the handheld device 112 through 3D space, thereby generatingoutput to control the robotic motion of the camera 102 and/or the crane104 and/or the dolly 106, while viewing essentially real time feedbackincluding the image captured by the camera 102, for example. The visualfeedback can be used by the operator 106 to improve the control of thecamera 102 and/or its supporting equipment, such as to record orbroadcast live action, or to record or refine camera moves, to name afew examples. In some implementations, camera settings such as focus,aperture, shutter speed, and/or zoom can be controlled in addition tocamera movement, to name a few examples.

Any of multiple techniques can be used for generating the output tocontrol the robotic motion of the camera 102 and/or the crane 104 and/orthe dolly 106. Examples involving one or more sensors mounted on oraround the handheld device 112 will be described below. In the presentexample, the system 100 includes one or more sensors 118 a-d mounted ina fixed arrangement around an area 120 where the handheld device 112 isto be used. In some implementations, one or more of the sensors 118 a-dcan include motion-detecting cameras and can optically register themovement of the handheld device 112 and/or of the operator 116, whereinthe output controlling the robotic motion of the camera 102 and/or itssupporting equipment can be generated using the motion-detectingcamera(s). For example, one or more motion-detecting cameras canregister that the handheld device 112 is moved in a forward direction inthe area 120, and this can generate an output that causes the dolly 106and/or the robotic crane 104 to move the camera 102 generally in aforward direction with regard to the scene 110. As another example, oneor more motion-detecting cameras can register that the operator 116makes a change in relative position to the device 112 (e.g., byextending the arms holding the device), and this can generate an outputthat causes a change in the position of the camera 102 and/or a changein one or more camera settings (such as focus, aperture, shutter speed,and zoom). As another example, one or more of the motion-detectingcameras can register that the operator 116 makes a predeterminedgesture, such as a hand gesture, for example, and this can generate anoutput that causes a change in the position of the camera 102 and/or achange in one or more camera settings.

In some implementations, the motion-detecting camera(s) can transmitvideo and/or related positional and movement information via one or morewired and/or wireless connection 122 to a computer 124. Here, thecomputer 124 can process video and/or related information to determinethe position and the movement of the operator 116 and/or the handhelddevice 112 as it is manipulated by the operator 116. Any kind ofcomputer device can be used as the computer 124, such as a personalcomputer or a server computer, to name just two examples.

One or more markers (such as objects with distinguishing shapes, colors,and/or light intensities, to name a few examples) can be placed on thehandheld device 112 and/or on the operator 116 allowing tracking by anoptical motion tracking system, for example. Optical motion trackingsystems can use data captured from sensors such as cameras totriangulate the 3D position of a subject and to determine its movement.In some implementations, video information can be transmitted from themotion-detecting cameras to the computer 124 which can process video andcan determine the movement and position of the device 112, primarilybased on calculated movement and positions of the markers, for example.In some implementations, video information can be processed at least inpart by systems at the motion-detecting cameras, and relevant movementand positional information can be transferred to the computer 124, forexample. Here, for example, movement and positional information can beinterpreted by the computer 124 to determine the movement and positionof the device 112 and/or the operator 116.

In some implementations, markers placed on the device 112 and/or theoperator 116 can be associated with other physical properties instead ofor in addition to light, such as sound, magnetism, or electromagneticradiation, to name a few examples, and the periphery of sensors 118 a-dcan be set up to detect one or more of these physical marker types.Here, for example, movement and positional information can betransferred by one or more of the physical sensors 118 a-d via theconnection 122 and can be interpreted by the computer 124 to determinethe movement and position of the device 112 and/or the operator 116.

In some implementations, the computer 124 can communicate with thecamera 102 and/or the automated crane 104 and/or the dolly 106, via oneor more wired and/or wireless connection 126, for example. Here, thecomputer 124 can control the camera 102 and/or the crane 104 and/or thedolly 106 using inverse kinematics robot control techniques, forexample.

In some implementations, the camera 102 can generate a live image of thescene 110 and can relay the image to the display device 114 associatedwith the handheld device 112 via the connection 126, thecomputer/transmitter 124, and a wireless connection 128, for example.Any wireless protocol can be used, such as the 802.11 series ofprotocols, for example. In some implementations, the live image can berelayed to the display device 114 from a transmitter connected to thecamera 102, for example.

In some implementations, the wireless connection 128 can be used totransmit movement and positional information from the handheld device112 to the computer/receiver 124. For example, in some implementations,one or more components of a system for determining the movement and/orposition of the device 112, such as movement and/or position sensors,can be located on the device itself. For example, a plurality of markerspositioned around the device 112 can be detected by sensors on thedevice 112 to determine its relative position and movement (as will befurther exemplified in FIG. 2). In some implementations, a hybrid oftracking systems can be used to determine the position and movement ofthe handheld device 112 and/or the operator 116.

FIG. 2 is a diagram of an example of device maneuvers 200 that can beemployed to control or otherwise cause robotic camera maneuvers 202. Asschematically illustrated by a relational arrow 204, one or more of thedevice maneuvers 200 being performed can cause one or more of the cameramaneuvers 202 to be executed, for example by robotic techniques asexemplified in the system 100.

In some implementations, the handheld device 112 can include one or moresensors 206. The sensor 206 can, for example, function as a positiondetector for determining the position and/or movement of the device 112.In some implementations, the sensor 206 can generate output that can beused to control the camera 102, for example by robotic techniques asexemplified in the system 100. In some implementations, the roboticmotion causes the camera 102 to emulate the handheld device 112 whilethe live image from the camera 102 is presented on the display device114.

The sensor 206 generating the output for controlling the robotic motionof the camera 102 can be any kind of sensor such as an inertia sensor,an optical sensor, an audio sensor, and/or a magnetic sensor, to name afew examples. In some implementations, markers or transmitters can bepositioned around the device 112 to provide a set of reference pointsfor the sensor 206, but these are not explicitly shown in the figure forclarity. Methods for determining the position and/or movement of thedevice 112 can use the sensor 206, sensing and tracking techniques usingfixed sensors as exemplified in system 100, and hybrid systems combiningmultiple systems and techniques, to name a few examples. Output forcontrolling the camera 102 can be generated by any exemplary methods.

One or more robotic maneuvers can be applied to the camera 102 bymaneuvering the device 112, for example. Here, from the point of view ofone situated in front of the display device 114, for example, the device112 can undergo a rightward or leftward movement 208. In the presentexample, the movement 208 can be detected (e.g., by the sensor 206) andan output can be generated that causes the camera 102 to undergo arightward or leftward movement 210, which in some implementations isproportional to the movement 208. As another example, the device 112 canundergo a forward or backward movement 212 and the camera 102 can becaused to undergo a forward or backward movement 214. As anotherexample, the device 112 can undergo an upward or downward movement 216and the camera 102 can be caused to undergo a upward or downwardmovement 218.

As another example, rotational maneuvers such as panning (or yaw),tilting (or pitch) and/or rolling can be applied to the camera 102 bymaneuvering the device 112, for example. Here, from the point of view ofone situated in front of the display device 114, for example, the device112 can undergo a panning maneuver 220. In the present example, themaneuver 220 can be detected (e.g., by the sensor 206) and an output canbe generated that causes the camera 102 to undergo a panning maneuver222, which in some implementations is proportional to the maneuver 220.As another example, the device 112 can undergo a tilting maneuver 224and the camera 102 can be caused to undergo a tilting maneuver 226. Asanother example, the device 112 can undergo a rolling maneuver 228 andthe camera 102 can be caused to undergo a rolling maneuver 230.

Two or more of the maneuvers previously described can be performedconcurrently in any of various combinations. For example, the device 112can undergo a tilting maneuver 224 while undergoing an upward motion216. In the present example, the combination of maneuvers can bedetected (e.g., by the sensor 206) that can generate output that cancause the camera 102 to undergo a corresponding tilting maneuver 226while undergoing a corresponding upward motion 218. In someimplementations, any combination of movements can be performed by thedevice 112 and applied to the camera 102.

In some implementations, certain maneuvers can be restricted. Forexample, it may be determined that the camera rolling maneuver 230 isundesirable. In the present example, the device rolling maneuver 228 canbe ignored (e.g., by the sensor 206) or by systems receiving generatedoutput (e.g., by the sensor 206). In the present example, if the device112 were to undergo a rolling maneuver 228 and a simultaneous backwardmotion 212, the camera can be caused to undergo a corresponding backwardmotion 214 but not a rolling maneuver 230.

In some implementations, movement of the device 112 can be determinedbased on location information (as determined by the sensor 206 and/orother methods exemplified in the system 100) and corresponding movementof the camera 102 can be scaled according to at least one predefinedparameter. For example, a parameter can be defined to specify thatforward movement of the camera 102 can be ten times greater (or smaller)than the controlling forward movement of the device 112. In the presentexample, a forward movement 212 of the device 112 often centimeters cantranslate to a corresponding forward movement 214 of the camera 102 ofone meter (or one centimeter). Different, similar or identical scalingparameters can be specified for each of the device maneuvers 200 and thecorresponding camera maneuvers 202, for example. Scaling parameters canapply to the position and/or the acceleration of the robotic camera 102and the controlling device 112, for example.

FIG. 3 is a diagram of an example of a system 300. In someimplementations, the system 100 can provide remote control to one ormore robotic and/or virtual cameras, for creating composite images, andfor recording and refining camera movement, to name a few possibilities.The system 300 can include a plurality of real and/or virtual systems,such as the system 100, a secondary system 310, and/or a virtual system320, for example. In examples below, it will be shown that one or moreof the systems can be controlled (e.g. by an operator or a computer)individually or two or more systems can be controlled simultaneously. Asanother example, it will be discussed that images generated by multiplesystems can be used for creating composite images, and/or that roboticor virtual camera operation can be recorded, recalled, and enhanced.

Here, for example, the primary system 100 can include the primaryrobotic camera 102 positioned in relation to the primary scene 110. Insome implementations, the operator 116 can control the primary camera102 using the handheld device 112 by methods, for example as previouslydescribed.

In some implementations, the secondary system 310 can include asecondary robotic camera 312 supported by robotic systems similar tosystems supporting the primary robotic camera 102, for example. Here,the secondary camera 312 can be positioned in relation to a secondaryscene 314, for example. In some implementations, the secondary camera312 can be a miniature camera, such as a lipstick camera, for example,and the secondary scene 314 can be a miniature scene. In someimplementations, the secondary camera 312 and scene 310 can be the samesize or larger than the primary camera 102 and scene 110.

In some implementations, the virtual system 320 can include a virtualcamera 322 and a virtual scene 324. The virtual system 320 can includecomputer components for performing its functions, such as a processor,and a memory and/or a hard drive for storing information such assoftware. The virtual scene 324 can include a computer-based 3D modelsuch as a solid or shell model, for example. The scene 324 can begenerated by any method such as using a dedicated modeling program or amodel description language, to name a couple of examples. The virtualcamera 322 can represent a viewpoint regarding the virtual scene 324,and can be designed to operate in virtual space analogously to how aphysical camera would in real space, for example.

In some implementations, the primary camera 102 in the primary system100, and/or the secondary camera 312 in the secondary system 310, and/orthe virtual camera 322 in the virtual system 320 can be controlledsimultaneously. For example, the operator 116 can cause the handhelddevice 112 to move on a path 330 a-b. In the present example, inresponse to the motion of the device 112, the primary camera 102 can becaused to move on a path 332 a-b, for example by robotic control methodsas previously described. In some implementations, the secondary camera312 can be caused to move, in response to the motion of the device 112,on a path 334 a-b, for example by similar robotic control methods. Insome implementations, the virtual camera 322 can be caused to virtuallymove, in response to the motion of the device 112, on a path thatcorresponds to zoom positions 336 a-b with regard to the virtual scene,for example as schematically illustrated in the drawing. In someimplementations, the control of the virtual camera can be performedusing methods that can be employed by a computer graphics engine. Here,for example, one or more of the cameras 102, 312, and 322 cansimultaneously travel along their respective paths 332 a-b, 334 a-b, andassume the zoom levels 336 a-b. Any or all of the paths can be modifiedcompared to the path 330, for example by translation or scaling.

In some implementations, the operator 116 can be in visual communicationwith any or all of the scenes, such as the primary scene 110 as viewedby the primary camera 102, the secondary scene 314 as viewed by thesecondary camera 312, and/or the virtual scene 324 as viewed by thevirtual camera 322. For example, visual feedback can be presented on thedisplay device 114 included in the handheld device 112. Visual feedbackcan include at least portions of images from one or more of the cameras102, 312, and/or 322, allowing the operator 116 to control any of thecameras individually, or allowing the operator 116 to control one ormore of the cameras simultaneously, for example.

In some implementations, the output that controls the robotic motion ofthe primary camera 102 can also control the second camera 312. Here, forexample, the primary camera 102 can be associated with the primary sceneor set 110 and can generate a live image from the primary set 110 at thecamera position 332 a. In the present example, the secondary camera 312can be associated with the secondary scene or set 314 and can beconfigured to generate a live image from the secondary set 314 at thecamera position 334 a. Both the primary camera 102 and the secondarycamera 312 can be placed in other locations based on locationinformation, for example. In some implementations, location informationcan be provided by the handheld device 112 and/or a computer model (ascan be provided by the computer 124, for example). Here, while theprimary camera 102 is moved from position 332 a to position 332 b andwhile the secondary camera 312 is moved from position 334 a to position334 b, at least portions of live images from both the camera 102 and thecamera 312 can be presented, for example at the display device 114 asshown by a composite image 340.

In some implementations, the output that controls the robotic motion ofthe primary camera 102 can also control the virtual camera 322. Forexample, the primary camera 102 and the primary scene or set 110 canoperate in parallel with the virtual camera 322 and the virtual scene orset 324. Here, the virtual camera 322 and the virtual set 324 can begenerated by a computer, such as a computer device associated with thevirtual system 320, for example. In the present example, the virtualcamera 322 can be associated with the virtual set 324 and can beconfigured to generate an image from the virtual set 324 at the cameraposition 336 a. Both the primary camera 102 and the virtual camera 322can be placed in other locations based on location information providedby methods such as the exemplary methods, for example. Here, while theprimary camera 102 is moved from position 332 a to position 332 b andwhile the virtual camera is moved from position 336 a to position 336 b(here, a forward motion), at least portions of live images from both thecamera 102 and the virtual camera 322 can be presented, for example atthe display device 114 as shown by a composite image 350.

The system 300 can be used for controlling, recording, recalling, and/orrefining camera movement, to name a few possibilities. Here, forexample, in the system 100, the initial camera location 332 a can beidentified for the camera 102 and the camera can be placed at thelocation 332 a. The path from location 332 a to location 332 b can becalculated for the camera 102 and the camera 102 can traverse the path332 a-b by robotic motion, for example. In some implementations, pathand/or location information can be provided via the handheld device 112.For example, the handheld device can register one or more key locations,such as the initial device location 330 a, the final device location 330b, and any other desired key locations along the path 330 a-330 b. Inthe present example, the system 100 can calculate the analogous camerapath 332 a-332 b, including the corresponding initial camera location332 a, the final camera location 332 b, and any other key locations,considering such factors as scaled movement and acceleration, forexample. Here, the path 332 a-b of the camera 102 can be determined by acalculation, as can be performed by the computer 124, for example.

In some implementations, the system 300 can be configured to identifyand avoid camera obstructions. For example, the system 100 can includean obstruction 360 in relation to the set 110. Here, one or moreobstructions, such as the obstruction 360, for example, can beidentified between the initial camera location 332 a and the finalcamera location 332 b. The system 100 can take the obstruction 360 intoaccount in calculating the path 332 a-b so that the obstruction 360 doesnot interfere with the camera 102. In some implementations, theobstruction 360 can be identified as part of a model of the set 110. Themodel can be a virtual or a real model that can be identified by thesystem 100 and referenced by the processing computer 124, for example.In some implementations, any or all of the systems 100, 310, and 320 cancause their respective cameras 102, 312, and 322 to undergo movement toavoid one or more obstructions associated with the system controllingthe camera, or associated with a model of the system, for example. Inthe present example, one or more systems not directly associated with anobstruction can cause a camera associated with the system to undergo amovement analogous to a movement taken by a camera to avoid anobstruction. Here, for example, the camera 102 in the system 100 canmove on the path 332 a-b to avoid the obstruction 360. Here, the camera312 in the system 310 can move on the analogous path 334 a-b, whether ornot an obstruction analogous to the obstruction 360 exists in the system310, for example.

In some implementations, camera movement can be determined by predefinedsequences, for example. Sequences can be a stored set of cameraoperations, such as camera movement and other camera settings such asfocus, aperture, shutter speed, and zoom, to name a few examples. Insome implementations, sequences can be defined with the use roboticcontrol software and/or virtual systems, and stored and performed by thecomputer 124 in the system 100, for example. In some implementations,sequences can be recorded while being performed in real-time, forexample as performed by the operator 116 using the handheld device 112.

In some implementations, the operator 106 can move the handheld device112 to specify a predefined movement. In the present example, output canbe generated by the device 112 and registered, for example by thecomputer 124. Here, path information can be recorded based on theregistered output before the camera 102 undergoes the predefinedmovement. In some implementations, the camera 102 can undergo themovement while recording.

Stored and/or recorded sequences can be modified and refined by thesystem 300. For example, the camera 102 can be caused to undergo apredefined movement based on path information, such as movement alongthe path 332 a-b. Here, an aspect of the predefined movement can beidentified for modification using the handheld device 112. For example,the tilting capability of the robotic camera 102 can be controlled bythe operator 116 using the device 112 while the camera moves along thepath 332 a-b. In the present example, one or more parameters associatedwith one or more aspects of camera movement and/or camera settings canbe modified using location information based on the device 112 and/orthe operator 116. In some implementations, the modified parameter(s) canbe recorded in association with the path information such that themodified parameter is taken into account in subsequent use of the pathinformation. Here, for example, the modified tilting aspect can bestored with the predefined path information and the camera 102 can becaused to undergo a combined set of operations at a subsequent time. Bythe exemplary methods, camera operation instructions can be defined,refined, and stored for subsequent use.

FIG. 4 shows an example of a process 400 for controlling a camera. Thecamera can be a robotic camera or a virtual camera, to name a couple ofexamples. In some implementations, the process 400 can be performed inthe system 100, the system 310, and/or the virtual system 320, forexample by a processor executing instructions from a computer readablestorage device. More or fewer steps can be performed; as anotherexample, one or more steps can be performed in another order.

The process 400 can include a step 402 for receiving locationinformation regarding a device (such as the handheld device 112, forexample) carried by an operator (such as the operator 116, for example).In some implementations, the device can include a display device (suchas the display device 114, for example) for continuously presenting alive image from a camera (such as the camera 102 or the camera 312, forexample) configured for being placed in a plurality of locations byrobotic motion. In some implementations, the device can include adisplay device for continuously presenting an image from a virtualcamera (such as the virtual camera 322, for example) configured formovement through virtual space by a computer processor, for example.

The process 400 can include a step 404 to calculate a new location forthe camera. In some implementations, the new location can be calculatedusing location information regarding the device, such as the locationinformation received in step 402, for example. In some implementations,scaling methods such as the described exemplary methods can be used inthe location calculations. The location calculations can be performed bya computer processor (such as a processor associated with the computer124, for example).

The process 400 can include a step 406 to place the camera in a newlocation. For example, the new location can be determined by a methodsuch performing the calculations in step 404. A robotic camera can beplaced in the new location by robotic motion, for example. In thepresent example, while the camera moves, a live image from the roboticcamera can be presented on the display device. As another example, avirtual camera can be moved through virtual space by instructions thatcan be executed on a computer processor, for example. In the presentexample, while the camera moves, a computer-generated image from thevirtual camera can be presented on the display device.

FIG. 5 is a schematic diagram of a generic computer system 500. Thesystem 500 can be used for the operations described in association withany of the computer-implement methods described previously, according toone implementation. The system 500 includes a processor 510, a memory520, a storage device 530, and an input/output device 540. Each of thecomponents 510, 520, 530, and 540 are interconnected using a system bus550. The processor 510 is capable of processing instructions forexecution within the system 500. In one implementation, the processor510 is a single-threaded processor. In another implementation, theprocessor 510 is a multi-threaded processor. The processor 510 iscapable of processing instructions stored in the memory 520 or on thestorage device 530 to display graphical information for a user interfaceon the input/output device 540.

The memory 520 stores information within the system 500. In oneimplementation, the memory 520 is a computer-readable medium. In oneimplementation, the memory 520 is a volatile memory unit. In anotherimplementation, the memory 520 is a non-volatile memory unit.

The storage device 530 is capable of providing mass storage for thesystem 500. In one implementation, the storage device 530 is acomputer-readable medium. In various different implementations, thestorage device 530 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device.

The input/output device 540 provides input/output operations for thesystem 500. In one implementation, the input/output device 540 includesa keyboard and/or pointing device. In another implementation, theinput/output device 540 includes a display unit for displaying graphicaluser interfaces.

The features described can be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or incombinations of them. The apparatus can be implemented in a computerprogram product tangibly embodied in an information carrier, e.g., in amachine-readable storage device or in a propagated signal, for executionby a programmable processor; and method steps can be performed by aprogrammable processor executing a program of instructions to performfunctions of the described implementations by operating on input dataand generating output. The described features can be implementedadvantageously in one or more computer programs that are executable on aprogrammable system including at least one programmable processorcoupled to receive data and instructions from, and to transmit data andinstructions to, a data storage system, at least one input device, andat least one output device. A computer program is a set of instructionsthat can be used, directly or indirectly, in a computer to perform acertain activity or bring about a certain result. A computer program canbe written in any form of programming language, including compiled orinterpreted languages, and it can be deployed in any form, including asa stand-alone program or as a module, component, subroutine, or otherunit suitable for use in a computing environment.

Suitable processors for the execution of a program of instructionsinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors ofany kind of computer. Generally, a processor will receive instructionsand data from a read-only memory or a random access memory or both. Theessential elements of a computer are a processor for executinginstructions and one or more memories for storing instructions and data.Generally, a computer will also include, or be operatively coupled tocommunicate with, one or more mass storage devices for storing datafiles; such devices include magnetic disks, such as internal hard disksand removable disks; magneto-optical disks; and optical disks. Storagedevices suitable for tangibly embodying computer program instructionsand data include all forms of non-volatile memory, including by way ofexample semiconductor memory devices, such as EPROM, EEPROM, and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,ASICs (application-specific integrated circuits).

To provide for interaction with a user, the features can be implementedon a computer having a display device such as a CRT (cathode ray tube)or LCD (liquid crystal display) monitor for displaying information tothe user and a keyboard and a pointing device such as a mouse or atrackball by which the user can provide input to the computer.

The features can be implemented in a computer system that includes aback-end component, such as a data server, or that includes a middlewarecomponent, such as an application server or an Internet server, or thatincludes a front-end component, such as a client computer having agraphical user interface or an Internet browser, or any combination ofthem. The components of the system can be connected by any form ormedium of digital data communication such as a communication network.Examples of communication networks include, e.g., a LAN, a WAN, and thecomputers and networks forming the Internet.

The computer system can include clients and servers. A client and serverare generally remote from each other and typically interact through anetwork, such as the described one. The relationship of client andserver arises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

A number of embodiments have been described. Nevertheless, it will beunderstood that various modifications may be made without departing fromthe spirit and scope of this disclosure. Accordingly, other embodimentsare within the scope of the following claims.

1. A system comprising: a first camera generating a live image of ascene, the first camera configured for being placed in a plurality oflocations by robotic motion; and a handheld device that includes adisplay device for continuously presenting the live image, whereinmovement of the handheld device causes the handheld device to generatean output that controls the robotic motion.
 2. The system of claim 1,wherein the handheld device includes a tablet device having a generallyflat configuration with the display device located on a main face of thetablet device.
 3. The system of claim 1, wherein the handheld deviceincludes an inertia sensor and wherein the output controlling therobotic motion of the first camera is generated using the inertiasensor.
 4. The system of claim 1, wherein the handheld device includesan optical sensor and wherein the output controlling the robotic motionof the first camera is generated using the optical sensor.
 5. The systemof claim 1, wherein the handheld device includes a magnetic sensor andwherein the output controlling the robotic motion of the first camera isgenerated using the magnetic sensor.
 6. The system of claim 1, furthercomprising: a plurality of second cameras mounted in a fixed arrangementaround an area where the handheld device is to be used, the secondcameras registering the movement of the handheld device, wherein theoutput controlling the robotic motion of the first camera is generatedusing the second cameras.
 7. The system of claim 1, further comprising:an automated crane on which the first camera is mounted, the automatedcrane configured for receiving the output and placing the first camerain any of the plurality of locations by the robotic motion.
 8. Thesystem of claim 1, wherein the first camera is associated with a firstset where the scene takes place and wherein the first camera generatesthe live image from the first set, the system further comprising: asecond camera associated with a second set, the second camera configuredto generate another live image from the second set; wherein the outputthat controls the robotic motion of the first camera also controls thesecond camera.
 9. The system of claim 8, wherein the second set is avirtual set generated by a computer and the second camera is a virtualcamera generated by the computer.
 10. A computer-implemented method forcontrolling a camera, the method comprising: receiving locationinformation regarding a handheld device carried by an operator, thehandheld device including a display device for continuously presenting alive image from a first camera configured for being placed in aplurality of locations by robotic motion; calculating a new location forthe first camera using the location information; and placing the firstcamera in the new location by the robotic motion while presenting thelive image on the display device.
 11. The computer-implemented method ofclaim 10, further comprising: identifying an initial location of thefirst camera; and calculating a path for the first camera from theinitial location to the new location, the first camera to traverse thepath by the robotic motion.
 12. The computer-implemented method of claim11, further comprising: identifying at least one obstruction between theinitial location and the new location; and taking the obstruction intoaccount in calculating the path so that the obstruction does notinterfere with the first camera.
 13. The computer-implemented method ofclaim 12, wherein the first camera is associated with a first set andthe obstruction is identified using a model of the first set.
 14. Thecomputer-implemented method of claim 11, wherein calculating the pathcomprises: determining a movement of the handheld device based on thelocation information; scaling the movement into a scaled movementaccording to at least one predefined parameter; and determining the pathso that the first camera undergoes the scaled movement.
 15. Thecomputer-implemented method of claim 10, further comprising: causing thefirst camera to undergo a predefined movement specified based on pathinformation; identifying an aspect of the predefined movement to bemodified using the handheld device; modifying a parameter using thelocation information, the parameter corresponding to the aspect of thepredefined movement; and recording the modified parameter in associationwith the path information such that the modified parameter is taken intoaccount in a subsequent use of the path information.
 16. Thecomputer-implemented method of claim 15, further comprising: registeringan output from the handheld device generated by the operator moving thehandheld device before the first camera undergoes the predefinedmovement, the operator moving the handheld device to specify thepredefined movement; and recording the path information based on theregistered output before the first camera undergoes the predefinedmovement.
 17. The computer-implemented method of claim 15, furthercomprising: receiving the path information before the first cameraundergoes the predefined movement, the path information received from anapplication program configured for specifying robotic camera control.18. The computer-implemented method of claim 10, wherein the firstcamera is associated with a first set and generates the live image fromthe first set, the method further comprising: associating a secondcamera with a second set, the second camera configured to generateanother live image from the second set; placing the second camera inanother location based on the location information, while presenting atleast portions of the live image and the other live image on the displaydevice.
 19. The computer-implemented method of claim 18, wherein thesecond set is a virtual set generated by a computer and the secondcamera is a virtual camera generated by the computer.
 20. A systemcomprising: a camera generating a live image of a scene; an automatedcrane on which the camera is mounted, the camera configured for beingplaced in a plurality of locations by robotic motion using the automatedcrane; a handheld device that includes a display device for continuouslypresenting the live image; and a position detector generating an outputthat controls the robotic motion based on movement of the handhelddevice, wherein the robotic motion causes the camera to emulate themovement of the handheld device while the live image is presented on thedisplay device.